In this paper, we use a linear program to empirically decompose fitted neural networks into ensembles of low-bias sub-networks. We show that these sub-networks are relatively uncorrelated which leads to an internal regularization process, very much like a random forest, which can explain why a neural network is surprisingly resistant to overfitting. We then demonstrate this in practice by applying large neural networks, with hundreds of parameters per training observation, to a collection of 116 real-world data sets from the UCI Machine Learning Repository. This collection of data sets contains a much smaller number of training examples than the types of image classification tasks generally studied in the deep learning literature, as well as non-trivial label noise. We show that even in this setting deep neural nets are capable of achieving superior classification accuracy without overfitting.

This paper presents an interesting idea, which is that deep neural networks are able to maintain reasonable generalization performance, even on relatively small datasets, because they can be viewed as an ensemble of uncorrelated sub-networks. Quality: The decomposition method seems reasonable, except for the requirement for the model and the sub-nets to achieve 100% training accuracy. While there are some datasets where this will be reasonable (often high-dimensional datasets), there are others where such an approach would work very badly. That seems to me a fundamental weakness of the approach, especially if there are datasets of that nature where deep neural nets still perform reasonably well. For a random forest, we have an unweighted combination of base classifiers, but it is a learned combination in the case of the decomposed sub-networks, and the weights are tuned on the training data.

However, we only ever hear of using AI to understand big data sets. This is because small data sets are usually easily understood by people, and applying AI to analyze and interpret them isn't necessary. These days, many businesses and manufacturers integrate AI into the production line, slowly creating data scarcity. And unlike big companies, many setups cannot collect massive training sets due to risk, time, and budget limitations. As most companies don't know how to benefit from AI application on small data sets correctly, they blindly apply it to make future predictions based on previous files.

Sample size always plays a role in data science, but there are certain instances where risk, time or expense will limit the size of your data: You can only launch a rocket once; you only have so much time to test a much-needed vaccine; your early-stage startup or B2B company only has a handful of customer data points to work with. And in these small data situations, I've found that companies either avoid data science altogether or they are using it incorrectly. One of the more common issues in applying AI is blindly relying on historical data for predicting future situations -- I call this "assuming the past is the future." A common example of this is when we assume the model that has worked so well for us in previous markets will work the same "magic" when we use it to launch products in a new market. The problem is, our new market -- the future -- is completely different from the past market, which leaves us with poor judgement, incorrect predictions, and lackluster business results.

In this paper, we use a linear program to empirically decompose fitted neural networks into ensembles of low-bias sub-networks. We show that these sub-networks are relatively uncorrelated which leads to an internal regularization process, very much like a random forest, which can explain why a neural network is surprisingly resistant to overfitting. We then demonstrate this in practice by applying large neural networks, with hundreds of parameters per training observation, to a collection of 116 real-world data sets from the UCI Machine Learning Repository. This collection of data sets contains a much smaller number of training examples than the types of image classification tasks generally studied in the deep learning literature, as well as non-trivial label noise. We show that even in this setting deep neural nets are capable of achieving superior classification accuracy without overfitting.